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ABSTRACT 

Metacognition, literally thinking about thinking, is a term 
used by cognitive psychologists to refer to our ability to monitor our own 
performance on cognitive tasks . The term also addresses the ability to assess 
level of knowledge and skill in a given domain. Behavioral and psychological 
researchers frequently solicit metacognitive judgments from research 
participants in the form of self-assessment survey items. However, evidence 
suggests that metacognitive judgments are often at odds with reality. The 
term metacognitive miscalibration is used to refer to this disparity between 
self-assessments and more objective measures of ability and performance. The 
authors have theorized a relationship between metacognitive miscalibration 
and underachievement in courses where many students enter the class believing 
they already know the material. This paper reports preliminary results from 
an ongoing study seeking to understand the relationship between metacognitive 
miscalibration and underachievement in a computer literacy course. Five 
figures include pretest quiz results, miscalibration and recalibration in the 
pretest, comparison of pretest and posttest quiz scores, comparison of 
improvement by recalibration group, and posttest quiz results. (Author) 
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ABSTRACT 

Metacognition, literally thinking about thinking, is a term used by cognitive psych ologists to refer to our ability to monitor our 
own performance on cognitive tasks. The term also addresses the ability to assess our level of knowledge and skill in a given 
domain. Behavioral and psycho logical researchers frequently solicit metacognitive judgments from research participants in the 
form of self-assessment survey items. However , evidence suggests that ourmetacognitive judgments are often at odds with reality. 
The term metacogn itive mis calibration is used to refer to this disparity between self-assessments and more objective measures 
of ability and performance . The authors have theorized a relationship between metacognitive miscalibration and 
underachievement in courses where many students enter the class believing they already know the material. This paper reports 
preliminary results from an ongoing study seeking to understand the relationship between metacognitive miscalibration and 
undera chivem ent in a computer literacy course. 



INTRODUCTION 

“Professor Smith, I want to talk to you about my exam. I’m 
really surprised at my grade. I mean, I come to class everyday. 
I really know this material. I took computers in high school. 
And I feltreally goodabout theexam. Ijust don’t understand 
how I could have gotten such a low grade. Is there any chance 
there was some kind of mixup in the grading or something? I 
just know I didn’t fail that test.” 

What causes students to maintain such a high opinion of their 
performance in the face of overwhelming evidence to the 
contrary? Is it possible that this phenomenon in some way 



impairs the student’s ability to learn? If so , might itbe possible 
to correct the student’s misconceptions and, thereby, improve 
learning? 

These questions form the basis ofan ongoing research project, 
originally proposed in Smith and Foltz (2000), examining the 
relationship between metacognitive miscalibration and 
underachivem ent. The goals ofthis research are to develop an 
instrument that will identify students who are highly 
miscalibrated, examine the relationship between miscalibration 
and course performance, search for ways to recalibrate 
students’ metacognitions, and, finally, determine whether 
recalibration improves course performance. 
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MET A COGNITIVE MIS CALIBRATION 

Metacognition is not to be confused with affect. Some early 
reviews of this work suggested including studies of students’ 
attitudes toward computers in the literature review. While 
affect may influence metacognition, whether students like 
computers or feel good about using computers, the subjectof 
many MIS educational studiesnot cited here, is not the subject 
of this research. 

Metaco gnition, literally thinking about thinking, is a term used 
by cognitive psycho logists to refer to our ab ility to monitor our 
own performance on cognitive tasks. The term also addresses 
the ability to assess our level of knowledge and skill in a given 
domain. Behavioral and psychological researchers frequently 
solicit metacognitive judgments from research participants in 
the form of self-assessment survey items. However, evidence 
suggests that our metacognitive judgments are often at odds 
with reality (Kruger & Dunning, 19 99). The term 
metacognitive miscalibration is used to refer to this disparity 
between self-assessments and more objective measures of 
ability and performance. 

Psychological researchers have several theories to explain 
metacognitive miscalibraton. At the cognitive level, the cue 
familiarity theory (Metcalfe et al,, 1993) essentially restates 
the old cliche, “a little knowledge is a dangerous thing.” When 
people have some k nowled ge of the d omain in question, they 
are likely to have higher metacognitive judgments than when 
they have no knowledge of the domain. At the social level, the 
above average effect (Dunning et al., 1989; Alicke et al., 1995) 
simply says that people have higher opinions of themselves 
than of others. These self-serving assessments tend to increase 
with the amb iguity of the trait being assessed (Dunning et al., 
1 989) and the levelof abstraction in the comparison (Alicke et 
al., 1995). 

Regardless of the source of metacognitive miscalibration, 
psychological and educational researchers have related degree 
of miscalibration to both prediction of performance on 
multiple-choice exams (Sinkavich, 1995) and actual exam 
performance (Shaughnessy, 1 979). There seems to be general 
agreement that competence begets more accurate 
metacognitions (Maki et al., 1 994; Kruger & Dunning, 1 999; 
Shaughnessy, 1979; Sinkavich, 1 995). One study suggeststhat 
gaining competence in the domain is the only way to correct 
metacognitive miscalibration (Kruger & Dunning, 1999). 

Of more practical concern is research suggesting that 
metacognitive judgments influence decisions to continue work 
on a problem (Metcalfe, 1998) and studying (Bjork, 1996), 
This being the case, improving metacog nitive judgments - 
metacognitive recalibration - should produce better results in 
courses where students are highly miscalibrated. However, if 
Kruger and Dunning are correct in their assertion that only 
domain competence will result in recalibration, then we are at 
an impasse. 



RESEARCH METHOD 

A 50-item multiple-choice quiz was developed as a 
comprehensive test of course knowledge. Questions covered 
eleven subject are as, with no fewer than four and no more than 
six items in each area. Ofthe five possible responses for each 
item, the last was “I don’t know,” and the remaining four did 
not include any obviously incorrect choices. (For example, 
when asked who is credited with designing the first computer 
mouse, Walt Disney would be considered an obviously 
incorrect choice.) Four metacognitive instruments were 
developed, soliciting students’ self-assessments oftheir course 
knowledge relative to their peers. 

Participants were students enrolled in two summer sections of 
a computer literacy course, taught by the same faculty 
member, A total of33 studentscompleted the course pretest; 
23 completed theposttest. The course pretest, administered on 
the first day of class, consisted of the first metacognitive 
instrument, followedby the multiple-choicequiz, followedby 
the second metacognitive instrument. The course posttest, 
administered on the last day of class, prior to the final exam, 
consisted of the third metacognitive instrument, followed by 
the multiple-choice quiz, followed by the fourth metacognitive 
instrument. Students were instructed to choose the “don’t 
know” response if they were unsure of an answer on the 
multiple-choice quiz. Students were not compensated in any 
way for participation and were not informed of either their 
individual performance or the aggregate class performance on 
the quiz. 

RESULTS 

Following the pretest, the multiple-c hoicequ iz was scored and 
number of correct, incorrect, and don’t know responses 
recorded for each p articipant. T he mea n and sta ndard deviatio n 
of correct responses was used to construct five groups of 
students: far below average, below average, average, above 
average, and far above average. These groups correspond to 
response options on the metacognitive instruments. The range 
of correct responses labeled as average was defined as one 
standard deviation, centered on the mean. Above average was 
defined as the 1.5 standard deviations beyond average; far 
above average as the final two standard deviations. Below 
average and far below average were similarly defined. 
However, due to the low mean value and the large standard 
deviation, there were no observations in the far below average 
group. There was only one observation in the far above 
average group; this was discarded as an outlier. Thisleft seven 
students in the below average group, eleven in the average 
group, and four in the above average group. Figure 1 shows 
the mean number of correct, incorrect, and don’t know 
responses in each of the three pretest groups. 

Item two on the first metacognitive instrument asked 
participants to rate their “knowledge of concepts and sk ills to 
be covered in the course,” relative to their classmates. 
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Response options were far below average, below average, 
average, above average, and far above average. The 
corresponding item on the second metacognitive instrument 
asked participants to rate “overall performance on the pretest,” 
again relative to their classmates. Figure 2 shows the nominal 
linerepresenting perfectcalibration and the mean response for 
each pretestgroup on both the firstand second metacognitive 



instruments. The distance between the nominal line and the 
actualmean represents the magnitude of miscalibration for the 
group. The average and above average groups were well 
calibrated, though their self-assessments were somewhat 
below their actual performance. As anticipated the below 
average group o verestimated their performance, though not to 
the degree expected. 



FIGURE 1 

PRETEST QUIZ RESULTS 

n = 22 




FIGURE 2 

MISCALIBRATION AND RECALIBRATION IN THE PRETEST 

n=22 
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At the end of the term, the posttest was administered and 
scores again plotted by pretest group. Overall, the class 
improved their quiz scores by an average oftwelve questions; 
the range was two to twenty. As illustrated in Figure 3, though 
all groups improved their quizscores, the below averagegroup 
showed the greates t overall impro vement, an average of fifteen 
questions. 



What we had hoped to see, of course, was a difference in 
improvement between those who recalibrated on the pretest 
and those who did not. Figure 4 shows mean pretest, posttest, 
and improvement scores, based on correct responses to the 
quiz, grouped by whether recalibration occurred during the 
pretest. Those who recalibrated on the pretest had lower 
pretest scores, on average, than did those who failed to 
recalibrate. However, by the end of the course, there is 
virtually no difference in quizscores between the two groups. 



FIGURE 3 

COMPARISON OF PRETEST AND POSTTEST QUIZ SCORES 

n=22 




FIGURE 4 

COMPARISON OF IMPROVEMENT BY RECALIBRATION GROUP 

n = 22 



Comparison of PreTest Recalibration Groups 
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Figure 5 

Posttest Quiz Results 

n=22 




Finally, posttest groups were formed in the same manner as 
pretest groups. Again, there were no students in either the far 
below or far above average groups. At the end of the course, 
ten students were below average, six average, and six above 
average. Figure 5 shows the mean number of correct, 
incorrect, and don’t know responses for each posttest group. 
It is interesting to compare Figure 5 to Figure 1. Notice that 
the number of incorrect responses for the below average group 
is much higher on the posttest than on the pretest, more than 
double. At the same time, the number of don’t know responses 
dropped for all groups. 

DISCUSSION 

Are any of these observations statistically significant? 
Unfortunately, errors in data collection resulted in our having 
fewer than half the observations we hoped to acquire over the 
course of the summ er term. As a result, we do not have enough 
data at this time to do the analysis we had planned for this 
conference. We are, however, still collecting data and hope to 
present a more meaningful analysis in the near future. 

Nonetheless, there are some encouraging observations to be 
made. Interestingly, and un expectedly, recalibration seem s to 
have occurred as a result of taking the multiple-choice quiz. It 
is difficult to explain why, in the absenceof any feedback on 
actual performance, students would recalibrate in such a 
dramatic fashion.Inthewordsof one author, “it looks like we 
scared ’em.” 

To confirm this, an analysis of the pretest data for the Fall 
2000 term was performed to determine if what appears to be 



recalibration is merely a test-retest phenomenon. Because the 
first and second metacognitive instruments are not identical., 
the possibility o fa test-retest problem seemed sm all. However, 
several students in the original study tailed to complete the 
second metacognitive instrument, saying it was a duplicate of 
the first (Those observations were subsequently discarded.) 
However, given the likelihood that participants perceive these 
two instruments to be the same, itseemed best to consider the 
possibility that what appears to be recalibration is nothing 
more than a test-retest problem. 

Two different sections of the computer literacy course, taught 
by two different faculty m embers, were used for this analysis. 
In one section, the same instrument packet and protocol as 
were used in the Summer 2000 term were used; 34 complete 
observations were collected. In the second section, the 
multiple-choice quiz was replaced by a questionnaire asking 
students to provide some background information for the 
instructor. In addition, rather than receiving the second 
metacognitive instrument, which refers to pretest performance, 
these subjects were asked to complete a second copy of the 
first metacognitive instrument. A total of 31 complete 
observations were collected from the second class. 

Recalibration scores were computed as the difference between 
the second self-assessment and the first. An ANOVA was 
performed to examine differences between first self- 
assessment, second self-assessment, and recalibration scores 
for the two classes. There was no significant difference in first 
self-assessment scores for the two groups; p = 0.357. 
However, both the second self-assessment and the 
recalibration means were significantly different; p < 0.001 . In 
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fact, 16 of the 34 participants who received the multiple- 
choice quiz recalibrated while none of the 31 participants in 
the other class changed their self-assessments. 

The above analysis supports the idea that recalibration is 
occurring simply through exposure to course concepts on the 
multiple-choice quiz. In addition, faculty who have 
administeredthequiz reportthat fewer students approach them 
about “testing out” of the co urse than in previous sem esters. 
Although we canno t quantify this, it does support the idea that 
students adjust their expectations of course content or their 
self-assessm ents of course knowledge as a result of taking the 
quiz. 

Second, either incorrect or don’t know responses may provide 
an alternative measure of metacognitive miscalibration, 
perhaps eliminating the need for the metacognitive 
instruments. Looking at Figure 1 with this in mind, the above 
average group is more miscalibrated than the average and 
below average groups on the pretest. While the above average 
students clearly know more than their peers, they still scored 
very poorly on the pretest quiz; they do not know as much as 
they believe they do and, in this sense, are highly 
miscalibrated. By the tim e of the po sttest, when all participants 
have been exposed to the same course material., the below 
average group is less well calibrated then the average and 
above average groups (Figure 5). This is consistent with 
cl aims that with dom ain com petence comes the ability to make 
more accurate self-assessments; competence begets 
metacognitive (re)calibration. 

Finally, while the study that triggered pursuit of this research 
(Kruger & Dunning, 1999) examined metacognitions at a 
micro level (judgments were solicited for each question), the 
currentstudy takes a more macro ap proach (a single judgment 
for the entire bo dy of kn owledg e). It is encouraging to see the 
same pattern of miscalibration coming from the macro 
approach as was seen in the micro approach. 

CONCLUSIONS AND FUTURE RESEARCH 

It is difficult to draw any meaningful conclusions from such a 
small data set. However, we do believe that instilling domain 
competence is notthe only path to metacognitiverecalibration. 
The comprehensive course pretest seems to be producing some 
recalibration, even without feedback on test performance. Only 
additional data, currently being collected, will tell if the 
recalibration is significant and whether recalibration leads to 
improved course performance. 

The results we have see n thus far, c ombined with the informal 
observation that the pretest reduces the number ofrequests to 
“test out” of the course, reinforce our belief that course 
pretests are worthwhile. Furthermore, pretests make 
comparison of pretest and posttest performance possible, 
providing a source of encouragement and a measure of 
effectiveness for the faculty. 



Plans for furtherstudy include several manipulations including 
providing feedback to students on pretest performance and 
providingcompensation, in the form of bonus points, for b oth 
pretest and posttest performance. In addition, for the second 
round of data collection adjustments may be made to the 
multiple-choice quiz in order to raise the mean score and 
reduce the standard deviation, in an attempt to populate all five 
student groups. 

REFERENCES 

Alicke, M.D., Koltz, M.L., Breitenb echer, D.L., Yurak, T.J., 
and Vredenburg, D.S. (1995). Personal Contact, 
Individuation, and the Better-Than-Average Effect. 
Journal of Personality and Social Psychology, 68(5), 804- 
825. 

Bjork,R.A. (1 996). Memory and Metamemory Considerations 
in the Training of Human Beings. Metacognition : 
Knowing About Knowing , first MIT Press paperback 
edition, Metcalfe, J. & Shimamura, A.P., eds. Cambridge: 
MIT Press. 

Dunning, D., Meyerowitz, J.A., & Holzberg, A.D. (1989). 
Amb iguity and Self-Evaluation: The Role ofldiosyncratic 
Trait Definition s in Self-Serving Assessments of Ability. 
Journal of Personality and Social Psychology, 57(6), 
1082-1090. 

Kruger, J. & Dunning, D.(1999). Unskilled and Unaware of 
It: How Difficulties in Recognizing One’s Own 

incompetence Lead to I nflated Se lf-Assessm ents. Journal 
of Personality and Social Psycho logy, 77 (6), 1121-1134. 

Maki, R.H., Jonas, D., & Kallod, M. (1 994) . The Relationship 
Between Comprehension and Metacomprehension 
Ability. Psychonomic Bulletin & Review, 1(1), 126-129. 

Metcalfe, J. (1 998). Cognitive Optimism: Self-Deception or 
Memory-Based Processing Heuristics? Personality and 
Social Psychology Review, 2(2), 100-110. 

Metcalfe, J., Schwartz, B.L., & Joaquim, S. G. (1993). The 
Cue-Familiarity Heuristic in Metacognition. Journal of 
Experimental Psychology: Learning, Memory, and 
Cognition, 1 9(4), 85 1 -860. 

Shaughnessy, J.J. ( 1 979). Confidence- Judgment Accuracy as 
a Predictor of Test Performance. Journal of Research in 
Personality, 1 3(4), 505-5 1 4. 

Sinkavich, F.J. (1995). Performance and Metamemory: Do 
Students Know What They Don’t Know? Journal of 
Instructional Psychology, 22(1), 77-86. 



6 



Proceedings of the 1 5 h Annual Conference of the International Academy for Information Management 




7 



BEST COPY AVAILABLE 




U.S. Department of Education 
Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 




EFF-089 (1/2003) 



